In recent years, many studies have been reported with regard to the super-resolution processing which estimates one high-resolution image from multiple low-resolution images having displacements (see Non-Patent Document 1). Various methods of the super-resolution processing, for example, the ML (Maximum-Likelihood) method disclosed in Non-Patent Document 2, the MAP method (Maximum A Posterior) method disclosed in Non-Patent Document 3, and the POCS (Projection Onto Convex Sets) method disclosed in Non-Patent Document 4 have been proposed.
The ML method is a method which defines an evaluation function as square error between the pixel value of a low-resolution image estimated from a presupposed high-resolution image and the actually observed pixel value, and obtains a high-resolution image by minimizing the evaluation function as an estimated image. In other words, the ML method is a super-resolution processing method based on the principle of maximum likelihood estimation.
The MAP method is a method which estimates the high-resolution image by minimizing the evaluation function which added probability information of the high-resolution image to square error. In other words, the MAP method is a super-resolution processing method which uses certain prior information regarding the high-resolution image to estimate the high-resolution image as an optimization problem that maximizes posterior probability.
The POCS method is a super-resolution processing method which generates simultaneous equations regarding the pixel values of the low-resolution image and the high-resolution image, and obtains a high-resolution image by solving the simultaneous equations successively.
All of the above-described super-resolution processing methods have the common features of presupposing a high-resolution image and estimating its pixel value for each pixel of all low-resolution images based on point-spread function (PSF) obtained from camera model from the presupposed high-resolution image so that these methods can search for a high-resolution image by minimizing the difference between the estimated value and the observed pixel value (the observed value) Therefore, these super-resolution processing methods are called reconstruction-based super-resolution processing methods.
As described above, the reconstruction-based super-resolution processing is formulated as the optimization problem of the evaluation function defined with respect to the high-resolution image. In other words, the reconstruction-based super-resolution processing is boiled down to the optimization problem of the evaluation function based on square error between the estimated low-resolution image and the observed image.
Since the reconstruction-based super-resolution processing is an optimization calculation with very many unknowns, in order to optimize the evaluation function of the optimization calculation, an iterative calculation method such as steepest descent method is often utilized. In such a case, it is necessary to calculate an evaluation function and differential of the evaluation function with respect to a high-resolution image in each iterative calculation. However, the cost of the iterative calculation is very large. Also, in each iterative calculation, it is necessary to conduct estimation for the pixels of all low-resolution images, and the calculation cost of estimation is also very large.
That is, in the reconstruction-based super-resolution processing, since calculation cost is very large, reducing of calculation cost becomes an important problem.
In order to solve this problem, the inventors of the present invention suggested a “fast method of super-resolution processing” disclosed in Japanese Patent Application No. 2004-316154 (Japanese Patent No. 3837575) from the viewpoint of reducing the pixel value estimating calculation number of times for calculating an evaluation function. That is, this fast method of super-resolution processing is characterized by implementing registration of multiple low-resolution images, setting the discretized points in a high-resolution image space and utilizing the mean pixel value of the pixels corresponding to the neighborhood of the discretized points. In this case, it was shown that a high-speed calculation is possible without losing estimation accuracy by considering the number of the pixels corresponding to the neighborhood of the discretized points as a weight.
In other words, in Japanese Patent Application No. 2004-316154, the inventors of the present invention suggested a fast method of super-resolution processing which utilizes an evaluation function that can reconstruct a high-resolution image without losing estimation accuracy by a little pixel value estimating calculations of low-resolution pixels.
More specifically, the “fast method of super-resolution processing” disclosed in Japanese Patent Application No. 2004-316154 is characterized by considering the pixels of multiple low-resolution images after registration as the pixels sampled at unequal interval within a high-resolution image space, dividing the high-resolution image space into multiple small areas, and utilizing the mean pixel value of the pixels within each small area. The evaluation function for such a small area is expressed by the following Expression 1.I=M| f−{circumflex over (f)}(xc,yc)|2  [Expression 1]
Where, the following Expression 2 holds.
                              f          _                =                              1            M                    ⁢                                    ∑                              i                =                1                            M                        ⁢                                                  ⁢                          f              i                                                          [                  Expression          ⁢                                          ⁢          2                ]            
Where, I represents the evaluation function for a small area, M represents the number of pixels within the small area, fi represents the pixel value of the i-th pixel within the small area, (xc, yc) represents the representative position of the small area, and {circumflex over (f)}(xc,yc) represents the estimated value of the pixel value corresponding to the representative position of the small area respectively. In the actual calculation, the sum of the evaluation functions for all the small areas becomes the evaluation function of the whole super-resolution processing.
Then, the inventors of the present invention further utilized the evaluation function disclosed in Japanese Patent Application No. 2004-316154 and suggested a fast method of super-resolution processing according to the present invention, from the viewpoint of speeding up the calculations of the evaluation function and the differential of the evaluation function with respect to a high-resolution image.